Analyzing Approximate Value Iteration Algorithms

In this paper, we consider the stochastic iterative counterpart of value iteration scheme wherein only noisy and possibly biased approximations Bellman operator are available. We call approximate (AVI) scheme. Neural networks often used as function approximators, in order to counter Bellman’s curse dimensionality. they operator. Because neural typically trained using sample data, errors biases may be introduced. The design AVI accounts for implementations with sampling errors. present verifiable sufficient conditions under which is stable (almost surely bounded) converges a fixed point To ensure stability AVI, three different yet related sets that based on existence an appropriate Lyapunov function. These function–based easily new literature. verifiability enhanced by fact recipe construction necessary also provided. show analysis can readily extended general case set-valued approximations. Finally, more circumstances, is, finding points contractive maps.